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ABSTRACT 

One of a series of papers developed as part of a 
project to improve planning and evaluation in community colleges, 
this working paper discusses the outcomes approach to evaluation in 
higher education and describes current projects utilizing outcomes 
measures in community colleges. Section I defines the outcomes 
approach as an analysis of the end products of organizational 
activities, cites prior attempts to create models and ccmcepts for 
outcomes research in higher education, and presents an outcomes 
typology that categorizes research and writing. Section II discusses 
the relationship of outcomes evaluation to community and junior 
colleges and describes different types of learner outcomes (i.e., 
concrete learner outcomes such as transfer, degree attainment, and 
employment and income; abstract learner outcomes such as cognitive 
and affective development; and social learner outcomes such as 
benefits to the local community anc! to society at large). Section III 
reviews the ways in which the outcomes evaluation approach is 
currently being used in the evaluation and operation of postsecondary 
institutions, focusing on the standardized testing and measurement of 
outcomes, attempts to refine the concept of outcomes in postsecondary 
education, and the use of the approach in institutional management 
and the creation of funding formulas. Finally, the last section 
summarizes the current issues in outcomes evaluation. (HB) 
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PREFACE 

This worlr M<?asuring Learner Outcomes is one of a series of papers 

resultin(, ^. year project to improve evaluation and planning in 

commun _ .he project is sponsored jointly by the Chancellor's 

Office ot cne Ca. .rnia Community Colleges and by the Western Association 
Accrediting Comiflission for Community and Junior Colleges. Project work is 
concentrated in California and Hawaii, the jurisdiction of the Western 
Accrediting Cormission. Support for the project is provided by conmunity 
colleges in these states, the two sponsoring aggencies, and by the federal Fund 
for Improvement of Postsiecondary Education (FIPSE). 

Project objectives include developing a clear statement of the responsibilities 
i-or evaluation and planning that are appropriate for state control agencies, 
accrediting commissions,, and for local community colleges. T nsions about the 
appropriate division of these responsibilities exist throughout the country. A 
long tradition of cooperation in California and Hawaii, however, has created a 
most congenial atmosphere in which to analyze and clarify the proper 
delineation of roles. 

Project s«^,dff also are developing a series of tools to improve the state-of-the 
art of evaluation and planning for community colleges. Beginning in the Fall 
1982, these tools have been introduced, used and assessed in a dozen workshops, 
self-study seminars, symposia, and problem-solving sessions conducted in 
California and Hawaii. These activities will continue through the Fall of 
19b4. While project work is being concentrated in the two states, it should be 
possiole to generalize the results to virtually any community college operation 
or governance structure in the country. 

Working Paper Ten was prepared earlier in the project as a basis to begin work 
on the use of outcome measures in community college planning and evaluation. 
This paper assesses the literature and existing efforts that deal with the 
measurement of learner outcomes in community colleges. Definitions and models 
for outcomes measures are discussed, first for higher education generally then 
for community colleges specifically. This is followed by an examination of the 
ways outcome measures are used for evaluation. The paper concludes with a 
summary of the current issues in outcome evaluation. 

This paper served as background information for a three-day symposium of 
leaaing experts from across the country that was held in December 1982, This 
symposium involved a thorough discussion of the topic and proceedings are 
available. Results of the symposium have since been used in project workshops 
and dre the Oasis for further project work to develop measures and strategies 
for actual use in college evaluation efforts. 

The reader will note that we, the project staff, have other responsibilities. 
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Introduction 

This paper is being presented as a component of the 
joint FIPSE project entitled Improving Community College 
Evaluation and Planning. It is intended to provide a foun- 
dation for the discussion of the following isr-ues: 

The strengths and weaknesses of the outccxner. approach 
to evaluation in higher education. 

Strategies for applying the outcomes approach to ccm- 
mirity colleges. 

The use of existing sources of outcome data in the 
evaluation and planning of California's community col- 
leges. These sources include the California Community 
College Chancellor's Office Information System, 

Information useful for the discussion of these issues will 
be presented in three sections. 

Section I of this paper will define the language of 
outcome evaluation and cite prior attempts to create models 
and concepts for outcome research in higher education, TTiis 
section will also introduce a typology that categorizes out- 
come evaluation research and writ\ng. 

Section II will relate outcome evaluation to community 
and junior coHeftes. The major types of tv^-year college 
outcomes will be identifif-d and literature that cxanines 
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them will be Cited. TVils section will focu5 on the methods 
and issues that surroijTd outcome research in two-year insti- 
tutions. 

Section III will review the ways In v^ich the outcopie 
evaluation approach is currentl y being incorporated in the' 
evaluation and operations of postsecondary institutions. 
These efforts include the standardized testing and measure- 
ment of outcomes, attempts to refine "the concept of outcomes 
in postsecondary education, and the use of outcomes as 
institutional managOTcnt tools and in the creation of fund- 
ing formulas. 

Section IV will offer a summary of the current issues 
in outcome evaluation. 
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Section I " The Outcomes Approach 
Definitions 

The term "outcanes" has become a part of the language 
of organizational analysis. Used broadly, it refers to the 

end products of orphan izational activities. In higher educa- 
tion the simplest and most common outcome studies use simple 
descriptive statistics to depict sucJi things as degrees 
awarded and average grade point averages of students. 

Studies that exiinine more complex outcomes in higher 
education have become common in the last 50 years. Various 
authors have drawn together this outcome literature into 
reviews that define outcomes in specific ways. For exauple, 
in Measuring Outcomes of C ollege . Pace (1979) used the term 
"outcome" to refer U) the findings of large scale "achieve- 
ment testing, alunni surveys, and studies of institutions." 
Achievanent tests docunent knowledge acquisition as an out- 
come. Alunni surveys utilize follow-up studies and longitu- 
dinal designs to ascertain wtiat oirtcomes have manifested 
over time in the lives of graduates, institutional studies 
do not usually focus on learner outcomes as such but exanine 
the organizational activities that, create learner outcomes. 

Felcinan and Nev/uomb (1969) reviewed and analyzed educa- 
tional outcanes but focunsed on different kinds of studies 
and favor a different definition. In a two-volune review 
that encompassed over 1 /;00 empirical Studies conducted 
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between the 1920 's and the 1960's, they advanced the concept 
of "Impact" to chafacten'ze the "chaosc or development or 
adaptation" brought about in students. In this case, out- 
comes refers to the cognitive and affgctive changes experi- 
enced by individuals during the time spent in school. 

dowen i\sn^) presented a consi.se but thorough defini- 
tion of the outconies of higher education , He delineated 
three major educational services and the outcomes associated 
with them. The first service he identified was instruction. 
The outcomes associated with it were learning and changes in 
himai traits. The second Si?rv ice identified was research 
and scholarship that result in the "preservation, discovery, 
and interpretation of knowledge, artistic and social criti- 
cisni, philosophical reflection, and advancement of the fine 
arts." The third service identified was public service and 
its outcomes were identified as improved public health, 
ngricultural prodvKitivity , and contributions to the solution 
of social problems. 

A more ge^icral definition of outcomes was offered by 
Lenning ct at (1977). In arriving at their definition, the 
authors did not use ©npirical studies but instead reviewed 
the theoretical 1 iterature of higher education. They exarw 
inod the concept of outcomas by reviewing models, typolo- 
gies, and taxonomies that have been advanced to identify 
them, eventually defining outcomes as "any results or conse- 
quences of an educational institution and its programs." 

11 
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This brnwl definition is intended to subsme the narrower 
ones, thereby opening the consideration of outcomes to 
inclixie. such areas as conwivtiity serv»ce, provision of tech- 
nology and cultural activities. Their work incorporated the 
concerns of authors and researchers in higher education and 
organizational theory. They accounted for as many higher 
education outcomes as possible, including those defined as 
"planned output" (Hoenack et al , 1974) , "ultimate conse- 
quences" (Robinson and Majak, 1967), "intended benefits" 
(Hitch, 1970) and "side effects" (Cook and ScioU, 1972). 
The authors proposed six basic questions to make their broad 
definition usable. A condensed version of these questions 
and the considerations they entail are presented in Figure 1 
(from Micek, 1980). 

Other writers use the term outcome to denote the pro- 
ductivity of educational institutions. The consideration of 
productivity requires the Unking of outpiits with inputs in 
order to define educational outcanes. Inputs are generally 
defined as resources entering into organizational activi- 
ties; outputs as the specific products of an organizational 
activity. When the two are linked, the considerations of 
•institutional efficiency, effectiveness, and performance 
arise. 

Peterson (1977) delineated a niinber of performance 
measures and assembled them into the fcyplogy of 
input/output ratios repre.sented in Table 1. Giving specific 
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SIX BA^IC QUESTIONS IMPORTANT FOR UNDERStANDlNG 
AN EDUCATIONAL OUTCOME AND THE ATTRIBUTES OR 
FACTORS ASSOCIATED WITH EACH 



WHAT ARE THE CHARACTERISTICS AND MAKEUP OF AN "EDUCATIONAL OUTCOME"V 

Form - the basic configuration of the outcome as it observed and/or 
HSiilured. Outcomes can be separated into products, events, and condi 
tions. 

fhAnnp <^tatu^ - whether the outcome results in maintait.;r.g (preserving, 
rerenishng repro uc ny. or stabilizing) or changing (modifying enrich - 
Tng, ^estru^iuring, or replacing) the existing condition or state of 
affairs. 

Focus - the basic, specific "what" that is maintained or changed to 
Z5^llitute the ou come of concern (knowledge, understanding skil s. 
attUudes! roles, certification status, jobs. incon«, social conditions, 
technology, art forms, and so forth). 

Nputralitv - although people attach positive or negative value connota- 
ifsnrlsHpectfrc oStcL^^ the generic concept of "outcome" is a neut^.l 
one separated from any inherent value status. 

Measurability - the case with which the ^"^nthrrs1rr3ffflcu?t°Jo 
measured. i;om e outcomes are easily measured; others are dirricuii lo 

measure. 

and US proSucer/facilitator through outputs and intermediary impacts 
( impact). 

WHICH INSTITUTIONAL RESOURCES AND ACTIVITIES ARE COMBINED, AND IN WHICH 

., waVs. to bring about the outcome(5) of concern? 

Produ cer/Facilitato r - the progranwnatic or functional activities of an 
WcT tiona l instit uTion or its components that P^-^^uce and facilita^^ 
are intended to produce and facilitate, particular educational outcomes. 
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FOR WHOM IS THE OUTCOME INTENDED, OR WHO ACTUALLY RECEIVED OR WAS 
AFFECTED BY IT? 



Audience - the persons, groups, orqani zations, comrnunUies, aggregations 
cTrpeopTe with common observable characteristics, activities or other 
entities that receive and/or are afrected by (or are intended to ireceive 
or be affected by) the outcome of concern. 

WHY WIU, OR DID, THE OUTCOME OCCUR? 

Intended/ Uninten ded - whether the outcome was designed or planned to 
occ'ur'oV whether iT just happened. Included are the positive, negative, 
or neutral value zonnotations attached to an outcome by different people 
and groups, and tne "exchange value" perceived for the outcome by its 
producer/facilitator. , 

WHERE WILL, OR DID, THt OUTCOME OCCUR? 

Functional Area - the functional areas within the various audience entities 
that are'beTng affected by (or that are meant to be affected by) the 
outcome, such as economic, educational /technological , political, and 
social /cultural /personal . 

WHEN WILL. OR DID, THE OUTCOME XCW^? 

Time - the time, or expected tire, of occurrence of an outcome (such as 
prior to graduation, more thai one year after graduation) and the duration 
or persistence of the outcome (how long it lasts). 
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TABLE 1 

A TYPOLOGY OF PERFORMANCE MEASURES 



\. OutCOfTK Mf«$arf (0) 
A QiMntitMivi? 

II Oukomc tffiocncy UfVO) 
A. Qu«ntfUtit« 



111 OutCOfnr tffrctiMCncsstfVG) 



Emmpit 

N umber of dCffWS «%Mrdrd 

Inttrtictien*! coitW«frct 
C4Mt avtra'ietcst ia>re» of 
grjdutftes 

Number of 6nif^tUtP^\ for 
number of de|i«ts 
A WAffc tMfc Kore orf grada/fMl 
for jvcrjfetB* score 



A QuantrUtfvr Ci^tfckgrt^pMl for g«^«-^ 

B OuIliUtjve Owt levf I of MitrBgc t«t icfl/cs 



I, Impact MtMurcv (I) 
A. QuarttiUtive 

B QujbUtiw 

II. ImpecttffttitftcyCln/l)* 

HI Impact Eff«tl¥tn<jiU/0) 
A. QiMntiUtiw 



EmmpU 

A ¥rrige ti me to lom ptete degree 
Avvregf gpm on lUndardiied 
tot from entry to gredioAion 

Coft/ivf ragffc time to co/npl^ 
Coctf unit of everage pin on 
iUnd«rdiiei tct 



A vrrai^ time Id comptetr^ 
degrcc/foe) fw »vf reg/ttime 
Avrrage i»<ni on sUmUrdlod 
teO/jpiil for ym. 

IV. PterforiMnceEffecti«enes»an/l/C)* 

A. Qu»ntJt*t(vf Cost/*verJtetiriiet» complftc 

<»rgrei^goil ^ 
5. QujIJtutive C^ever*g*t«t«core gain^oel 

•or co«fc-^eneflt ffficcncy or effectiMmeM 
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definitions to outcome and impact, he described U-i -^ures 
as follows; 

"(Table D provides a simplified model of qualita- 
tive and quantitative outcone measures. It is evi- 
dent here that impact measures differ from outcome 
measures in trying to assess the effects of an edu- 
cational wcperience in terms of gains, as compared 
with final output. Impact and outcome measures can 
be viewed in terms of the cost of .resources to 
achieve them as input/output or e fficiency ratios 
for outcomes Cln/0) or impacts iln/l). Alternative- 
ly, outcome or impact measures can be viewed in 
terms of their relationship to some standard of in- 
tended goal CO: as an outcome effectiveness (0/G) 
or an impact effectiveness (I/G) measure. Further- 
more, since performance measures are intended to re- 
late inputs to outputs as well as compare them to 
some standard goal , perf orman ce effectiveness meas- . 
ures c^ be expressed as 'the inputToutput ratio for 
either inputs or outcomes compared to the goal s or 
standards established for them. Thus, outcome- based 
performance effectiveness measures are designed as 
IN/O/G, and impact-based performance effectiveness 
measures are shown as In/I/G. In essence, impact 
based performance effectiveness measures are analo- 
gous to cost-benefit measures." (pages 4-5) 

Linking outcancs to inputs in a differenL way, Astin 
(1977) was concerned with the difference between the value 
or level of the input and the value or level of the output 
(outcome). This relationship is not a ratio but an additive 
relationship; hence the term "value-added." As a psycholo- 
gist Astin uses the tenn to denote the cor- tri but ion of 
higher education to an Individual's intellectual , emotional 
»»d social life. In other words, value-added can refer to 
the cognitive, affective, psychological and behavioral jjai.ns 
that a student enjoys because of his exposure to higher edu- 
cation. 

16 
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Alt.prn,»t,iveiy, an economic definition of value-added 
war> utilized by Breneman and Nelson (»981): 

"The essence of value-added is a comparison between 
a stifdenfa situation or prospects for the future 
upon entering coUege with his situation or pro*- 
pects at the end of his enrollment. The rate of re- 
turn adds the costs of generating those benefits 
into the calculations. Technically, this approach 
determines the rate at which future income resulting 
from the education (the value-added in terms of in- 
come) must be discounted to equal the cost c»f pro- 
viding the education," 

This definition emphasizes the economic outcomes of educa- 
tion in which the cost of the education is weighed against 
the benefits. 

It is clear that current concepts of outcomes in higher 
education imply a broad range of effects. Outcomes can be 
classified in categories that vary from specific impacts on 
Individuals to broad socio- etoncmic phenomena. A second 
dimension for the classification of outcomes is concreteness 
to abstractness. For example, the Intel lectuar refinement 
or analytical skills enjoyed by a student are substantive 
outcomes of higher education but are abstract when ccxnpared 
to the concrete outcome of the degree the student was 
awarded or the income h» may enjoy as a result of his 
increased capabilities. A third dimension for the classifi- 
cation of oiitcomes Is time; outcomes may be immediate or 
they may surface only after months or years in the life of 
Lhn individual or society. Much of the equivocation of the 
term "impact" revolves around this dimension of time. For 
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some, impact means immediate outcomes, for others it means 
ultimate consequences. 

'fhe three dimensions jur>t described can be used to 
classify the outcomes of hisKer education. These dimensions 
can be sumarized as: , 

- the individual /social dimension 

- the concrete/abstract dimension, and 

- the dimension of time. 

Two of these three dimensions can be assembled into a typol- 
cjgy that is useful for the classificaUon and analysis of 

•-i 

educational outcomes. S\ 

■ \ 

An Outcomes Typology 

The typology presented in Table 2- has been constructed 
for the purposes of this paper. It incorporates the 
individual /social dimension and the concrete/ abstract dimen- 
sion Of outcomes. The form of this typology requires that 
the range of concrete and abstract outcone values be dicho- 
tomized. Material and atheoretical outcomes (e.g. achieve- 
ment scores, cofnrntnity service, income, degrees awarded) are 
concrete. Outcomes that are non-material or depend on 
theoretical constructs of disciplines such as psychology and 
philosophy are considered abstract (e.g. changes in the 
affective danain, increase in the hijnani.'5m of society). The 
individual/ social dirrrension is already dichotomous. For the 
sakf' of simplicity and utility the dimennion of time is not 
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dlrecUy incorporated into this framework. T^^e considera- 
tion of tinie, or . when outcomes are manifested, will be 
included as needed. 



TABLE 2 

A TYPOLOGY OF EDUCATIONAL OUTCOMES 



CONCRETE 



ABSTRACT 



INDIVIDUAL 
SOCIAL 



j — r 

Type 1 


Type 2 


Type 3 


Type 4 | 



1 c' 
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The eoipirical literature on outcomes in hig^^r educa- 
tion tends to deal with only one type of outcone at a time, 
usually Type 1 or Type 2 (the effects on individuals). The 
conceptual literature may deal with one type or any combina- 
tion of types. In the overview of the literature that fol^ 
low.s, consideration will be given to the broad conceptual 
approaches to outcomes that include two or more types of 
outcomes. 

An Overview of the Concepts and Models of CXJitccmes in Higher 
Education 

This overview is highly selective in its content. The 
information sumarized here was cliosen for two purposes; 
first, to illustrate the diversity of approaches to identi- 
fying outcomes, and second, to provide examples deemed use- 
ful for the FIPSE project. 

Some of the material contained in this subsection is 
taken fron the work of Lenning (1977). Under the sponsor- 
ship of the National Institute of Education (NIE) and the 
National Center for Higher Education Management Systems 

i 

(NCHEMS) the author apsembl <1 "over 80 previous attempts to 
structure educational outcomes and related concepts." Many 
of these reports were made available on a lijnited basis; 
they appeared in the proceedings of conferaoces, in institu- 
tional publications, or in other primary sources that are 
not readily available. As a result, the NCHEMS survey Is an 

20 
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invaluable secondary source of the conceptual literature on 
educational outcomes. 

•me Lenning survey was a precursor to another NCHEMS 
publication titled A Structure for the Outconies of ^ 
secondary Education As was stated 

previously, the authors of this later publication defined 
outcomes very broadly/^ The definitional model presented in 
the publication is depicted in Figure 2 . The model depicts 
three dimensions of outcomes: audit^nce, typ^ of outcome, 
and time. Figure 3 gives more detailed information on the 
"type of outcome" dimensions. Of interest to the FIPSE pro- 
ject are the economic and Njman characteristic outcomes; 
these Type 1 and Type ,2 outcomes (both concrete and abstract 
\individual outcones^ pr,e camensurate with the FIPSE project 
focus on learner cunc^^^- The "time" dimension is detailed 
in Figure 4. 



14 



Figure 2 

DIAGRAMMATIC OVERVIEW OF THE NCHEMS OUTCOMES STRUCTURE 



AUDIENCE 
DIMENSION 

(Who or what is 
intended to. or actu- 
ally does, receive or 
be aHected by ihe 
outcome'-') 



A Knowiedfl*. Tcchnolo<|y. 
antf Art Porm Outcome 
occurring at Time 1 for 
in<j'vi(juais/Groups 
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Af> Economic Outcorr^e 
occurring at Time 7 fcf 
Interest Based Communi" 
ties 



ividual/ \ \n\tt%s\-^^%t6 f Geographic 



Group Clients 



Communittes ) Based 

I Communities 
I 

Economic Outcomes 

i 

I 

I 

-r 

I 
I 
I 

Human Characteristics Outcomes 

I 
I 



I 
I 

Knowiodgt, Technology, and 

An form Outcomes 
I 

i 

I 

i 

I 
I 



i 



Retource and Servic* 

Provision Oulcomes 
• 
I 
I 
I 

T 

i 

i 

I 

\ 



Aggregates 

o< People 




TlMEDIMENSIO>J 

(When is th^ out- 
come expected to 
occur, Of when does 
it occur?) 



A Knowledge. Technology, 
and An Form Outcome 
occu"*ng at Time 2 for 
Aggrefates o( People 



TYPE OF OUTCOME DIMENSION 
(What basic entity Is, or4s Intend^ to be, maintained or changed?) 



Figure 3 



CODED LISTING OF THE SECOND- AND THIRD-LEVEL SUBCATEGORIES 
FOR EACH FIRST-LEVEL CATEGORY OF THE TYPE-OF-OUTCOME DIMENSION 



Cod^NumMM tftNty Oilino Mf.nl.inr.! or Ch.ngrd 




1000 BCOf^OUlCOUrCOMiS 
y \ 10 Cconon-.Jc Acr*j» 

M?0 f cortomfc Fmnunuy. Artafiiatmiiy ind S^cuftly 
1 \ y) [ncumm and SUndard o» I wing 
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An earlier fCHEMS outcome variables ider»tif ication pro- 
ject was conducteti by Micek and Wal lhaus (1973)- The out- 
comes they delineated, virtually all Type 1 and Type 2, are 
represented in Figure B • Although this list is smaller 
than the previous variables list, its simplicity and 
specific category headings make it easy to apply. 

Another conceptual approach to outccmes in higher edu- 
cation is the formulation of institutional goals. Goals 
stand as the intended outcomes of higher edLication and cover 
all four types of outcomes: individual to social, concrete 
to abstract. Goal delineation is a central part of at least 
three lands of institutional evaluation: effectiveness 
evaluation. , ^icicncy evaluation, and accreditation. Of 
these thre. , only accreoitation ha-j been mentioned pre- 
viously. All academic accrediting agencies rely upon stated 
institutional goals to assess the appropriateness of insti- 
tutional processes and the adequacy and use of institutional 
resources. 

The central importance of goal delineation in the 
evaluation of institutions and their outcomes has generated 
a large body of literature on the goals of higher education. 
This literature rarely contains distinctions between two- 
year 'rind four-year postsecondary education and generally ^ 
treats higher education as a wh;.>le. For example, the Carne- 
gie Commission (1973), devoted an entire volune to the dis- 
cussion of the broad goals or purposes of higher education 
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Figure S 
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MUpnntirfromMicekand Ualihaus {IW3. pp. 31-41). 
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for the years 1976 to EOOO. Five major purposes or "ewl 
objects" of higtw education were specified and the 
processes that lead to these ends were spelled out. Figure 6 
contains a stmnary of these goals and processes as 
abstracted by Lenning et al ('977). 

One of the few efforts that de*U directly with the 
goals of two-year institutions is the Educational Testing 
Service's Conminity Collei.e Goals Inventory (CCGI). The 
Inventory is designed o help^ community colleges define 
their educational goals, establish priority among those 
goals, and give direction to their present and future plan- 
ning. The list of the CCOr Outcome Goals Is presented in 
Figure 7 . It can be seen that the list contains all four 
types of educational outcomes. It ranges from such concrete 
individual outcomes as vocational /technical training to 
abstract social outcomes such as huniiniSi, al truism » and 
social criticism. 

The final t^JO concepts of outcomes to be discussed in 
this section also deal exclusively with the functions of 
two-year institutions. The first is the Florida 
ComniDity/Junior College IRC Taxonomy of Community Service. 
Nickens (1976) reported that in 197« the Interinstitutional 
Research Council (IKC) in Florida used a modification of the 
Delphi consensus- rendering technique to fxjll community col- 
lege leaders. The object was to delineate exactly what con- 
stitutes corimLTity Service for a community or junior col- 
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. Figure 6 

THE. CARNEGIE COMMISSION'S PURPOSES OF HIGHER EDUCATION* 

A. PROVIDE INDIVIDUAL STUDENTS WITH EDUCATION AND DE'fELOPMENTAL GROWTH 

1. Providing Broad Ledrning Experiences (General Education) 

2. Providing Specialised Academic and Occupational Preparation 

3. Assisting Academic Social izaticip 

4. Providing Interesting and Stimulating Campus Environments 

5. Providing Advisory and Counseling Support 

6. Providing Time to Assess Options and Make Choices Before Having to 
Make Commitments 

B. ADVANCE HUMAN CAPABILIT, SOCIETY AT LARGE 

1. Bringing About Research Advances and Dtvelopments 

2. Providing Service to Off-Canipus People and Organizations 

3. Finding, Assessing, and Placing Talent 

4. Training Ski lis 

5. Providing Cultural Information and Opportunities 

C. ASSIST THE REST OF SOCIETY TO PROVIDE EDUCATIONAL JUSTICE AND OPPORttfttl.-TlES 

1. Developing A4equate Hunbers of Open-Access and Other Places Offering 
Postsecondary Education 

2. Developing Special Programs, Including Those. That Are Remedial and 
Cultural 

3. Providing Essential Financial Support to Students 

D. PROVIDE SUPPORT FOR PURE SCHOLARSHIP, ARTISTIC CREATIVITY, AND THE 
ENHANCEMENT OF CULTURAL HERITAGE 

1 . Providing FacilHies 

2. Providing Personnel 

3. Providing a Favorable Climate 

E. PROVIDE AND STIMULATE EVALUATION OF SOCIETY THAT AIMS FOR SELF-REMEWAL 

1. Providing Freedom for Such Evaluation 

2 Providing Opportunities for Such Evaluation 

3. Providing Reasonable Rules of Conduct for Such Evaluation 



•Abstracted from Carnegie Cofwnission (1973, pp. 13-67) 
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EDUCATIONAL TESTING SEFiVlCE'S 
LIST OF C0M1UNITY COLLEGE GOALS 



OUTCOflE GOALS 



PROCESS 'GOALS , 



General Education 
Intellectual Orientation 
Lifelong Learning 
Cultural/Aesthetic Awareness 
Personal Development 
Human ism/Altruism 
Vocational/Technical Preparation ■ 
Developmental/Remedial Preparation 
Community Services 
Social Criticism 



Counseling and Advising 

Student Services 

Faculty/Staff Development 

Intellectual Environment 

Innovation 

College Community 

Freedom 

Accessibility 

Effective Management 

Accountability 
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lege. Results frofn this effort arc prcsentwJ in Figure 8 . 

r 

This concept is not concerned with individual outcomes (Type 
1 and 2) b^j^ instead the focus is on the social outcomes 
provided by the community colleges (Type 3 and 4), 

The second commtvtity-college-oriented concept is an 
input/outcome model, advanced by Alfred and Ivens CI978)- 
Their model, depicted in Figure 9 , was constructed to indi- 
cate the relationship between community college inputs, pro- 
grams, and outputs. In choosing e)Cdniples Npf outputs for 
their model, the authors used primarily Type 1 learner out- 
comes. This emphasis, although incomplete, iS appropriate. 
Most of the outcome research in community colleges focuses 
on these kind:^ of outcases. 

As the use and appl ication of outcome data become more 
ccjmmon, models that depict the methods by which these data 
can be applied to administrative decisions are developed. 
One such model has been constructed by the Learning and 
Retention ConsortiurQ CURC) and appeared in the LARC Program 
Guide (LARC, 1982). The Consortium is a group of fifteen 
California community colleges that have jointly developed an 
Assessment/Placement System that is used to assist students 
in developing the collcfte program that best suits their 
needs, basing the program choices in part on the students' 
rirnedial program outconieSp Appcundix A contair\s two models 
tha^ represent the assessfne/it/placenent system of the Con- 
sortitn. 




Figurt 8 

THE FLORIDA COMMUNITY/ JUNIOR COLLEGE IRC TAXONOMf 
FOR COMMUNITY SERVICES* 



1.00 Initructional Services 

1.10 General -Cultural Services 

1. IT Community and Civic Affairs 

1J2 Family Life 

V. 13 Leisure Time and Recreational Activities 

1.14 Personal Health 

1.15 Cultural Heritage and Enrichment 

1.20 Occufdtional Services 

1.21 Development of General Attitudes and 
Skills for a Career 

1.22 Development of Specific Attitudes and 
Skills for a Career 



2.00 Noninstructional Services 

2.10 Coordination 

2.11 Individuals 
2.1Z Groups 
2.13 Agencies 

2.20 Consultation 



2.21 Consultation with Individuals 

2.22 Consultation with Groups 

2.23 Consultation with Agencies 

2.30 Research and Development 



3.00 Facility Services 



•Abstracted fro<n NickenS (1976, pp. 13-10). 
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The next section of this paper examines, some of the 
current literature on the four types of conwunity college 
outcomes. In the cases where research on outccjoies is dis- 
cussed the review will not beconcernfl4 with the. actual 



findings bii^ instead will examine the current approaches to 
docunenting outcomes* 



W<^rJ.n ng Paper No. 10 

Section II - A Review of Community College Outcome Litera- 
ture, 

Introduction 

Outcane studies in commi^nity colleger most often exam- 
ine the extent to which institutions are carrying out their 
stated missions and functions. This, implies that outcome 
research focuses primarily on the intended outcomes of 
institutions. Such studies have become increasingly impor- 
tant over the last ten years and this importance can cer- 
tainly be related to the increasing scrutiny'of the commun- 
ity colleges' role in society. 

The unique characteristics of community colleges make 
the measurement of outcomes more difficult than similar 
research in senior institutions. Community colleges have 
comprehensive educational programs including transfer educa- 
tion, vocational education, continuing education, remedial 
education, and ' community services. Outcome measures which 
may be appropriate for one portion of the educational pro- 
gram may be completely inappropriate for others. Even the 
comprehensiveness of community colleges varies greatly 
, because of the various program mixes which are designed to 
meet local needs. Obviously, the needs of a small, isolated 
rural commirity differ substantially from thoae of a large 
inner city commuQity. The ch2Mracteristics of the enrolled 
stixlents in such colleges also varies subst>ntually , partic- 

2fl 
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ularly in the proportions of minority and low socioeconomic 
students* In addition to the wide diversity or students who 
enroll in community collies as a result of the open door 
policy, the commun'^ty colleges have also attracted several 

groups of students in<\luding the elderly, the handicapped,' 

\ 

and re-entry students, ^oth male and female. A further com- 

v 

plication is the mix of p^rt and full-time students. All of 
this diversity in educational programs and student and com- 
munity characteristics must\be taken into consideration when 
outcomes are to be defined ahd measured. 



The role of diverse servii^e that has been adopted by 
community colleges is summariz<^ in the state priorities of 
the California Community Collegei^* Board of Governors. These 
priorities emphasize access, programs, and services as major 
concerns. 

Open access to higher education has been one of the 
long-standing ideals of community colleges. The California 

Board of Governors has articulated tfie ""responsibility to 

\ 

provide prograns that ensure equal access to postsecondary 
education for all adults without regard to race, ethnic or 

national origin, sex, age, dis<)bility, of\ prior educational 

\ 

status," As an outcome, the ideal of access for the popula- 
tion as a wl-jole is guaranteed by law and 6Br\ be considered 
an abstract social benefit. 

Concrete benefits to individuals are emphasized by com- 
mitments to progr.jns and services. Six programs and 
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services are listed in the Board's statement and five of 
them offer specific outcomes fbr individuals: transfer edu- 
cation, preparation for employment, student support ser- 
vices, remediation and continuing and community education. 
These learner-oriented offerings have been the mainstay of 
commLnity colleges and they mirror the current priorities of 
commtrity colleges in general (Cross, 1981). Because of 
this common emphasis on benefits to individuals, most out- 
come studies focus on Type 1 outcomes. 

Type 1 - Concrete Learner Outcomes 

Three kinds of Type 1 outcomes will be reviewed here. They 
are the transfer of students, the degree, attainment of ccjm- 
munity college students, and the employment and income of 
community college graduates. 

The Transfer of st udents 

TVie problem of differentiating and typing transfer stu- 
dents in order to understand transfer outcomes was taken up 
by Willingham. Reinhart (1977) summarized WilUngham's 
classification of transfers: 

1. Articulated Vertical Transfer. Students moving 
directly from parallel, articulated programs in a 
two-year college into the upper division of the pro- 
gram in a four-year college. 

la. Articulated Vertical Transfer in Specialized 
Career Fields. This is a specialized case of Item 
1, applying to 5tudents whose associated degrees, by 
plan, are both entries to technical employment and 
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specialized degrees required for entry to an upper- 
division progran. J' 

2. Nontraditional Transfer. Two- and four-year 
college transfers who do not follow the usual pat- 
terns, including adults who have been out of college 
for some years and those involving external or ex- 
periential stixlles aiwi other situations in which the 
prior studies may not be valid for assessment pur- 
poses. 

3. Reverse Transfer. Students transferring from a 
four-year to a two-year college. 

^. Open Cbor Transfer. Transfers from one two-year 
college to another for a variety of reasons. 

5. Double Reverse Transfer. Those reverse 
transfers who return to a. four-year college. These 
individuals may be in normal transfer or in occupa- 
tional prograns at the two-year college, and may 
change their field in the process. 

6. Vocational to Changed Major Transfer. These are 
individuals transferring from a career progran in a 
two-year college to i elated but different bac- 
calaureate progra?ns in » four- year collie. 

7. Upside-Down Curriculun Transfer. This classifi- 
cation includes individuals that transfer into 
"upside-down" degree programs that exist in some 
four-year colleges. Sometimes involving a degree in 
"General Studifts," these degrees are structured to 
provide mostly general education courses, management 
studies, or other general studies that come after 
technical training in two-year colleges, (pages 39- 
J<0) 



These complexities prompted Reinhart to conclude: "Research 
results and policy assimptions are different for the several 
types of transfer. Therefore, assessment measures and cri- 
teria recamnendations must be b;»ncd on the Specific type of 
transfer." 

39 
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This conclusion of Reinhart's was echoed by Renkiewicz 
et al (1 992) . in a. .stud.en.t _Qutcome study that sampl ed over 
11,000 students from California's Los Rios Community College 
District. The transfer studenta exanined in the stixiy were 
classified as reverse and lateral transfers, with subclas- 
sifications of completers and non-completers. 

With the exception of several California Postsecondary 
Education Commission reports authored by Dorthy Knoell, 
(California Postsecondary Education Commission; 1976, 1979, 
1982), virtually all studies of commmnty college transfers 
depict outcomes dealing with Rcinnart's first kind of 
transfer student- the articulated vertical transfer. For 
exanple, in a study of the California Community Colleges, 
Kissler (1980) reported on the number and condition of stu- 
dents moving from a two-year to a four-year institution. 
While this focus on the upward movement of students is 
appropriate, given the mission of many two-year institutions 
to prepare for baccalaureate-granting schools, the picture 
of the community college transfer function remains incom- 
plete without an acccjmmodation of the several opportunities 
for transfer that community colleges provije. A broad rede- 
finition of tiie transfer function could assist researchers 
In their attompts to depict the wide range of transfer out- 
co(nes of community college** 

The intricacies of accurately typing transfer students 
and the variability of the conditions that affect transfers 
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irake outccxne assessment in Ihia area a challenging undertak- 
ing. Institutions condijcting self-studies that exanine the 
relationship between local conditions and the transfer func- 
tion will be pursuing important information. Holmstrom and 
Di5.conti {197'0, in their atud^ on transfers for the Ameri-- 
can- Council on Biucot ion, have stated that corunon student 
background factors, "about which we know a great deal, are 
less important determinants (of successful transfer) than 
experiences at the junior college, about which we know con- 
siderably less." 

Degree At taifwent b^^ Comnun ity College Students 

Outcome studies may exanine the success of students in 
thf? attainment of a decree. As will be discussed later, the 
changing coranunity college population makes this research 
problenatic. It has even been claimed that many community 
college students ate "cooled OMt", eventually being turned 
away fron degree attainment and any further advanced study 
(Karabei , i972) . However, when undertaken, degree-attainment 
research is most of fen local Institutional research or sta- 
tistical analysis of national data. Most local research, 
usually institutional self-studies, includes information on 
how many certificates and associate degrees have been 
granted in a given period of time. Research with a national 
scope, more often than not, focuses on the success of oom- 
iTVirity coll'fge transfer students in attaining a bachelor's 
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decree . j 

The nationat studies arc generally the most sophisti- 
cated and most widely distributed. These studies make use 
of databases created frocn large-scale longitudinal research 
on students. The two common sources for these data are the 
American Council on Education's follow-up studies and the 
National Longitudinal Study of the High School Class of 1972 
(NLS 72). Various researchers have used longitudinal data- 
bases differently when seeking to compare the degree attain- 
ment of transfer and native students. For exanple, Folger, 
Astin, and Bayer (1970) used simple statistical controls 
such as SES and student ability when comparing tte degree 
attainment rate of the two groups. Trent and Medsker (1968) 
and Astin (1972) used slightly i«ore sophisticated categories 
when they controlled for SES, ability, and aspiration. 
These studies and their 3tati5>ical methods, although more 
than ten years old, represent the state-of-the-art in 
resea; th on bachelor's degree attainment rates. 

Local studies that depict associate, degree attaiment 
rates use Simple descriptiv* statistics in their tallies, 
local researchers may feel th« need for student attribute 
controls that help explain why some students attain degrees 
and others do not. Until recenr.ly, attribute variables that 
accommodato the idiosyncracies of the changing community 
college population have not existed. Some changes occurred 
uAien Sheldon (1981) created attribute categories that were 
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tht 41rfct product of inttrvlcwlii« and M9tsslf« eoMity 
eoU^it tturt4f>ts. Hit "iiirlfnt prototypei* w caUtorlM 
that type sturtcnta according to how Uity art Mpaeifd to um 
tha covMiity coUffffS. Eighiaan eatagorftta or prototypta 
arr spMlfird; arvcn prototypes ar* uac<1 for typing tranaftr 
studenta, five are uaed for typing voeational atudcnta, and 
sii are uaed for typing "apaelal intareat* students. The 
prototype5 arc listed here: 

- Trnnafer students: 1. fU\l«tiiie transfer 2. part-tiaie 
trMisfer 1. undiaeipHnad tranafer 4. tachnieal transfar 9. 
intareoUigiata athlete 6. financial aid aeakar 7. aipadi^er 

• Vocation?!! students: 1. Progrffi eaiplet«r 2. job aedBer ^* 
.fob vw'*^^'' '« career chenger 5, Hecnaa aaintainar 

- Spteial inurest ttudenU: 1. Xeiavr sklllt itMlant 2. 
adueaiiofi ateker 3. art and eulttra a t wda w t «. 
ea^lortr/eiperiMntar 5. baaic akllU atudant 6, XtUX'tX 
transfer. 

Mhlle the uaefulneaa of thaaa new aaiagorlas have 
to be ftiUy aiplorad, they aay help taidaraoora the ecMpleii* 
ties of research on degree attainaent. 

iltudiaa of the degree-grenting Aaiotion of comity 
oeUegea have been inportant baeauae of tlia l aniawty U 
e^ta degree atlAi^raaent with aehievwent and ptraiatofiof io 
«rhooi . Notrver, the difficulty of aouatlng dagrte atUlw* 
mmt with aeadaiic eehieviMent or parsiatanee is highlftghUd 
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the direct product of interviewing and assessing coinmunity 
college students. His "student prototypes" ar^ categories 
that type students according to how they are expected to use 
the community colleges. Eighteen categories or prototypes 
are specified; seven prototypes are used for typing transfer 
students, five are used for typing vocational students, and 
six are used for typing "special interest" students. The 
prototypes are listed here: 

- Transfer students: 1. full-time transfer 2. part-time 
transfer 3. undisciplined transfer 4. technical transfer 
5. intercollegiate athlete 6, financial aid seeker 

7. expediter. 

- Vocational students: 1. Program completer 2. job seeker 
3. job upgrader 4. career changer 5. license maintainer. 

- Special interest students: 1. leisure skills student 2. 
education seeker 3. art and culture student 4. explorer/ 
axperimenter 5. basic skills student 6. lateral transfer. 

While the usefulness of these new categories have yet 
to be fully explored, they may help underscore the complexi- 
ties of research on degree attainment. 

Studies of the degree-granting function of community 
colleges have been important because of the tendency to 
equate degree attainment with achievement and persistence in 
school. However, the difficulty of equating degree attain- 
raent with academic achievement or persistence is highlighted 
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by the diverse aspirktions of these different kinds of stu- 
dents. For example) the vocational Job seekers are 
described as those who "attend college only long enough to 
learn vocational skills that will permit them to attain a 
semi-skilltd to highly-skilled job/' These students, and 
many others in like categories, may persist sufficiently to 
achieve their own ends without degree attainment. Even the 
most solidly academic student, the full-time transfer, may 
very well view degree attainment as superfluous to his or 
her needs. 

The next step in the assessment of degree attainment, 
as well as other outcomes, must accomodate this more 
thorough understanding of the community college population. 
In the absence of this information, the use of degree 
attairment as an outcome measure will continue to be prob- 
lematic. 

Employment and Income 

In the last decade community colleges have increased 
their emphasis on providing vocational /occupational oppor- 
tunities for their students. This increas«d onphasis has 
drawn the attention of those interested in community col- 
lege outccfTies. A large nimber of local studies and several 
national research efforts have fbcussed on Job attainment 
and income ns output measures. While the findings of these 
stufJies differ, the methods- and the problfins surrounding 
the methods- remain fairly consistent. 
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In local and regional efforts, follow-up studies have 
become the usual method for assessing the outcomes of 
employment and income. The Southern Regional Tducation 
Board reported that, in 1980, 98t of the 8^1 institutions 
they surveyed were planning follow-up studies. Also, the 

I 

nunber of studies conducted had gone from zero in 1976 to 5^ 
in 1979 (Southern Regional Education Board, 1981). Two rer- 
sons for undertaking these studies were cited most often- 
they were intended to assess: 

1. "College graduates employed, unemployed, em- 
ployed part-time, employed in jobs related to their 
majors, employed in- state" and 

2. "Salaries of jobs, career potential of jobs, 
type of employer, (and) how the job was found."' 
(page i) 

A typical follow-up study that addresses several of 
these question* was conducted and issued by the Maryland 
State Board for Conwunity Colleges (1981). The instrunent 
used in the study is contained in Appendix B. 

The principal problem of the study is sunmed up in a 
sentence in the surrmary section of the Maryland study: "The 
response rate anonfc those wtio received the questionnaire was 
'17 percent". 

The problem of bias from low response rates was common 
in virtually all of the follow-up studies exanined for this 
review. The metfiod of dealing with the problem varied con- 
;5iderably, however. In the most thorough studies, samples 
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of non- respondents were taken and telephone follow-ups were 
used to determine if non- respondents differed from respon- 
dents on key^ vari^les.\^ This kind of thoroughness was 
uncommon and /'generally, ^ocal researchers did not arttend to 
the problem of low response^ rates. Instead, they drew lim- 
ited conclusions, from the information that was originally 
received, 

A second problem with local follow-up studies is find- 
ing meaning in the results. This problem is rooted in the 
fact that local studies are individual in format and con- 
tent. This has two effects: first, the studies cannot be 
combined into an aggregated picture of community college 
oatcomes; second, 'in the absence of aggregate information 
there is no external norm against which an institution can 
measure its performance. For example, a study may indicate 
that a specific percentage of vocational students are find- 
ing work in their chosen area. But, without some standard 
norm for student placement, the outcome statistic itself is 
not necessarily meaningful. Finding meaning in follow-up 
study data is, to some extent, dependent on knpwing how well 
comparable institutions are performing. The /development of 
some standardization for local follow-up ' studies may 
represent the next important step in local community college 
outcome evaluation. ; 

A third issue confronts local institutions wanting to 

/ 

use follow-up st»Jdie3 for the assessment of employment and 
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income; will the benefits of the study justify the costs? 
Cost/benefit is important to consider in light of the prob- 
lems of validity with low response rates and unclear meaning ■ 
of the data. The expense of follow-up studies, particularly 

the more cocnplicated longitudinal designs, may be difficult 
to Justify. 

Large-scale national studies of community colleges- 
effects on employment and income are leas common but gen- 
erally more wide' y known. Tho most recent of these was 
released by the Brookings Institution (Brenemati and Nelson, 
1981). Originally intended to be an econanetJ'ic study of 
coiun^^ity college financing, the authors found that, within 
the conmi^itycol lege- s evolving environment, "questions of 
finance will become increasingly entangled in questions of 
institiit.onal mission and pur{»se..." liecause of this, the 
authors direct attention to the assessment of the outcomes 
implicit in institutional purposes, especially those out- 
comes related to employment and incoric. 

For their analysis the authors utilized datn from the 
National Study of the Hi|',h School Class of 1972 (NLS 72). 
W authors explain the importance of large-scale national 
data in the assess/nent of outcomes. TTiey claim that such 
infonnation is more representative of national characteris- 
tics than any local follow-up study could be.' Furthermore, 
the database contains personal characteristics such as fam- 
ily income. aRe. race, and sex that can be utilized in the 
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analysis. They state: "AlthouRh the natiomil stixlies and 
the data on which they are based are not perfect, they are 
more suited to answering the questions that we sec as most 
important..." 

In addressing questions about employment and inccwe, 
the authors used multivariate linear regression to ascertain 
the relationship between community college attendance and 
job and wage outcomes. Using U? predictor variables they 
were able to account for between 3.1 and 24,0 of the varia- 
tion in the "labor market outcome" dependent variables (i.e. 
wages, occupational status, and weeks employed). 



It seems apparent 



that a major limitation of the use of 



national databases and multivariate techniques is that they 
leave a large amount of variation in outcomes unexplained. 
Furthermore, they provide no information on local conditions 
that could guide administrators in the decision-making pro- 
cess. The large amount of variance left accounted for in 
such large scale studies may reinforce the tendency of com- 
mmity college researchers to rely on snail-scale follow-up 
studies. (For a more technical discussion of the problems 
, of multivariate analysis in college output studies, see 
Astin [19771, pages 263-366). 

Another study investigated the outcomes of employment 
and income using inter-regional data. Wilms (197'4) under- 
took a survey project to compare the effectiveness of public 
and private vocational training. His method, wliich has been 
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widely criticized, was to follow up on graduates of t\^-year 
public and private programs and document their success in 
finding the kind- of work they were trained for. In this 
effort, ds in the Brcneman and Nelson stixly, emphasis was 
placed on contrasting the "labor market outcomes" of public-- 
commLTiity collegps with the outcomes of a comparable group. 
Breneman and Nelson chose high school graduates without com- 
m»*iity college training as the comparison group; Wilms chose 
to canpare graduates of proprietary vocational programs with 
commmity college vocational graduates. This emphasis on 
comparative analysis makes a study more complex, particu- 
larly for a local researcher w ^ limited access to non- 
community college'data. However, without some kind of com- 
parison, research findings do little more than describe pro- 
. ^ram effects. Such findings do not have the power to demon- 
strate the value of programs versus other educational or 
work alternatives. 

in a different att-^pt to examine employment outcomes, 
the National Center for Educational Statistics has created 
the Vocational Kducation Data System (VED3). In this sys- 
tem, co»rununity colleges are required to report the results 
of follow-up on the employment status of graduated voca- 
tional education students. Colleges report low returns of 
information from their former students, usually below 30J. 
It ic questionable whether nationally aggregated data will 
provide val id and useful information. Requiretl follow-up of 
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anployera al.«K) yielded very low returns and has now bcer» 

discontinued. 

In general , the problems that exist in the assessment 
of etnpioytnent and income are the problens of survey and 
longitudinal research as a vAole. Low response rates, miss- 
ing data, sampling difficulties, and , in the case of local 
studies, the absence of comparisons between groups all work 
to limit the state-of-the-art. 

Type ? Outcome s 

Abstract learner outcomes, or Type 2 outcomes, have 

been characterized by Aatin (1977) a» being cognitive or 
affective. The cognitive outcomes of postsecondar> educa- 
tion include the development of mental abilities such as 
reasoning and logic. Affective outcomes include the 
developnent of values, attitudes, morals, aspirations and 
self-concept. T^ie assessment of these outcomes can be 
approached using either psycholoftical information which dei>. 
icts internal traits, or by using information about observed 
behaviors. With these catcsories, Astin, Panes, and Creger 
(1967) classified outcomes as depicted in Tabic 3 • 
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Cognitive and affective changes in postsecondary stu- 
dents were also of interest to Feldman and Ncwcomb (1969). 
They reviewed and reported on more than 1500 empirical stu- 
dies, most of which were involved with docunenting the Type 
2 outcomes of colleges and universities. In Volune 1 of 
their work the author. delineate two of the methodological 
problems that continue to delimit the state-of-the-art of 
Type 2 outcome studies: the first problem is the difficulty 
of controlling for student inpits when measuring outcomes; 
the second problem is the difficulty of inferring changes in 
the psychology of the student from scores on instruments. A 
closely related - problem that the authors do not directly 
address is the problem of controlling for normal personal 
growth that is not a product of the education institution. 

The first problan of controlling for input when assess- 
ing output is a central concern of any attempt to measure 
what educational or personal values are added by an institu- 
tion. In order for value-added to be assessed, all of the 
relevant input variables must be identified and accounted 
for. The difficulties of doing this with psychological 
variables may preclude all but the simplest Type 2 research. 
'aIso, the interaction effects that occur between the indivi- 
dual inputs and the environment may undermine the validity 
of the outcome. For exanple, a large sprawling suburban 
community coUege envlrorvient may have psychological effects 
on specific individuals that an urban envircrjnent might not. 
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As these interaction effects become, subtler and increase in 
ninber, the possibility of controlling for them and drawing 
any general izable conclusions about postsecondary value- 
added effects becomes more difficult. 

The second problem is an instance of the larger issue* 
of construct validity in psychological measurement. In the 
Feldman and Newcomb example, the process of maturation is 
anbiguous enough to disallow the kind of definition neces- 
sary for precise and repeatable testing. For exanple, in 
responding to questions about values, freshmen may answer 
with certainty and seniors may answer tentatively. Does 
this change indicate that the seniors are becoming less 
rigid and more thoughtful , or does it indicate a growing 
uncertainty in the older students? Rescsarchers who want to 
assess maturation and other psychological development must 
be careful to address this issue of construct validity; the 
instruments they choose must provide data valid enough to 
supply a basis for conclusions about the trait in question. 

The third problem is a broader issue that actually sub- 
sumes the previous two problans. It is the problem of how 
to demonstrate cause and effect in docimentiig educational 
outoomes. This issue is especially important in the con- 
sideration of Type 2 outcomes because of the ambiguities 
thiTt surround their measurement. Pace (1979) considers the 
question in the following way: . 
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"When researchers write about the impact of college 
on students, they are asking questions about cause 
and effect. .Is the particular behavior or condition 
they observe really caused by the college? Could it 
be caused by or attributed to some other event or 
circumstance— to family background, for example, or 
I.Q., or simply to the normal process of growing up? 
We know that students in the twelfth grade are tall- 
er and heavier than students in the eighth grade; 
but we also know that this gain in height and weight 
was not caused by going to high school. That, of 
course, is an obvious exanple of conditions which 
are associated without having any cause and effect 
relationship. This association is what statisti- 
cians warn of when they remind us that correlation 
does not prov^ causation. . .If one asks the question 
'What?' rather than 'Why?, there are a lot of simple 
answers — clear, straightforward, and consisteni- over 
time. Do stixients learn anything in college? Yes. 
Do they themselves believe that they have made pro- 
gress toward such ends as critical thinking, acquir- 
ing a body of facts and knowledge of a special 
field, personal and social development, tolerance, 
broadened literary acquaintance, and so on? Yes." 
(pages 5-6) 

This attitude about the assessment of outcomes in gen- 
eral and Type 2 outcomes in particular should encourage 
local commurity college researchpr.;. Straightforward and 
careful description of Typp 2 outcomes can, when compared 
with non-student data, help build a body of information use- 
ful for docunenting the psychological impacts that coaimunity 
cor.f^ges have on students. 



Type 3 aiid Type 4 Outcom es 

While the FIPSE project is specifically concerned with 
assessing learr)er outcomes, it is worth briefly noting that 
several kinds of community colleges outputs can be called 
social outcomes* These are the concrete and abstract bene- 
fits thnt accrue to the local community and to society at 
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large. 

Type 3 outcomes are those spec (f ic and concrete effects 
that conrriLriity col leges have on the local community. Such 
effects can be economic or serv ice- rel atcd . Economic out- 
conea include local taices paid, local goods and services 
purchased, construction undertaten, workers trained, and 
payroll administered. Service-related outcomes iriclude the 
provision of facilities for social, cultural, and rccrca- 
tional progrsns. These outcomes dttd methods for their 
assessment are dttailed in Condu cting Cotmiunity Impact St^H 
dies, A Handbook for Comnunit y Colleges (Arinijo, Micek, and 
Cooper, 1978). The Handbook offers advice on how community 
impact data can be gathered usir^ internal sources such as 
institutional records ar¥l using external sources such as 
citizen's groups, employers, civic leaders, and social 
agency leaders. It also offers the best current examples of 
instriments that can be used to survey external constituen- 
cies. 

Type u outcones arc the abstract benefits that cojimun- 
ity colleges offer to society as a whole. S(jch offerings 
include opportunity through open access to higher education, 
egalitarianlsm increased cultural richness, and social inno- 
vation. Tyiesc oi/tcomes arc virtually impossible to measure 
and are difficult to docuTient in any way other than through 
general st)eculntioh. Because of this, these intended bene- 
fits of education have been held more as ideals rather than 
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having been assessed as outcomes. 

Although there are no studies aimed at trying to mcsas- 
ure these intended benefits, a study was undertaken to 
determine whether or not these ideals are held by those who 

are associated with the community colleges. The Educational 
Testing Service (ETS) field-tested its Community College 
Coals Inventory (CCGI) in early 1979 by administering it to 
a diverse group of over 4,300 community college students, 
faculty and administrators. The CCGI format prompts respon- 
dents to rank order twenty potential goals as to how impor- 
tant any one goal is and how important it should be within 
the community colleges. The inventory contains five goals 
which, if achieved, could be considered Type 4 outcomes: 
human ism/ altruism, social criticism, innovation, freedom, 
and accessibility. 

When Cross (1981) analyzed the CCGI field-test data, 
the result was, in part, an assessment of current commitment 
to Type 4 outcomes. The thesis that emerged from the 
analysis was as follows: 

"...the late 1970 's and early 1980*s represent a 
plateau between two periods of high energy and a 
sense of mission in the coawnunity colleges. The old 
ideals that sparked enthusiasm and the sense of com- 
mon purpose in the community colleges have receded, 
and new ideals have not yet merged to take their 
place." (page 113) 

This thesis wns grounded in data derived from the only 
standardized instrument that :Tieasures commitment to Type 4 
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ouicomes. Althoufch a substantial body of sophisticated 
analyticdl and coojectural literature has addressed these 
kinds of goals nnd their attainment (eg. Pincus, 1980), the 
state-of-the-art in the measurement of Type 4 outcomes has 
yet to move beyond this rough rank-ordering of goals. Ihtil 
some "psyc^)ometry,of society" which tests for social health 
and developnent is created , the assessment of Type 4 out- 
comes must rely on rough measures and analytic conjectures. ■ 

ruction Til - So(ne Current Projects Utilizing CXAtcomc Meas- 

ures • 

Intr oduction . 

The asscsancnt of post secondary outcomes has been a 
topic' that has received a great deal of attention in the 
last twelve years. C>ie of the first foruns in whiqh this 
topic was explored was a 1970 seminar at U.C, Berkeley's 
Center for Research and Development in Higher Education. 
During the proceed inp^s, Fredrick Balder ston, a Professor of 
Business Administration at, Berkeley and a writer on issues 
in higher f?viucation, spoke about the need to develop stra- 
tegies for outcome assessment: 

"We have b«jnped hard into ttie questions of outputs 
and their measurement because, among other things, 
we arc seeking now to link the resources used tb the 
results achieved— in other words, to link inputs 
with outp;'t:-5. It turns out that in the long history 
of concern aSnjut the processes and activitipn of 
education, we have achtevcd a very imperfect grasp 
of the nature of its results. Now we are ha^fim to 
tackle the problem:-, of output definition under 
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lorcpd dfaft...The job we h^ve to do is urgent, im- 
portant, and contruvprsial . If we hnd time, we 
might do well to sympathize with ourselves for tak- 
ing it on." (Baldcrston, 1970) 

Since that seminar, efforts to perfect the grasp on the 
nature of educational outcomes have continued. From 1970 to 
1976 the majority of the effort was in the definition and 
conceptualization of the general area. Research, both 
national and local, has continued as well. Although Pace 
(1979) has shown that studies exanining outcomes have 
emerged for at least half a century, most of the local and 
national research on community college outcomes and their 
causes has been released only since the mid-1970's. 

The concern^ about postsccondary outcomes in general 
continues to evolve in the form of several extant projects. 
Some of these projects are designed to make use of outcomes 
in the management of higher education while others represent 
new developments in the assessment of learner outcomes. 

Outcomes and Management 

Linking outcomes to management decisions is the focus 
for a project funded by the W.K. Kellogg Foundation and 
administered by the National Center for Higher Education 
Management .Systems (NCHEMS). The project coordinates seven 
public institutions, each of which is gathering outcome 
infonnation for use in various administrative decisions. 
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While each institution bc^an its own individijal program 
in January, 1992, there are three basic approaches being 
developed. The first appro^h Cinphasi7.es the use of out- 
comes for strategic planning; the second approach utilizes 
outccnes in the acadeMic program review process; tKe third 
approach involves assessing outccxnes of student services^ 
As a result of these efforts, NCHEMS anticipates being able 
to disseminate administrative models and research findings 
to other institutions, to state coordinating and governing 
boards, and to accrediting agencies. 

Another management-oriented outcomes project is the 
[.)erformance funding program of the Tennessee Higher Educa- 
tion Commission, This program encourage outcome evaluation 
by offering institutions the opportunity to win points that 
are redeemable in the form of budget increases* Points are 
/jwarded if the institution is simply willing to undertake 
specific outcome assessments; larger point awards are avail- 
able if the institution provides evidejnce that they have 
improved their own performance on a^ given variable or are 
substantially superior to comparable insti}:utions on a given 
vari^^ble. 

The five variables that can be assessed to win funding 
piints are: 1, Program accreditation 2. Progrc'jm field 
evaluation 3. Edacotional outcomes ^. Instructional improve- 
ments, and Planning for instructional improvonenti; 

Ihree of these five variables directly involve the use of 
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learner outcones. These three are the program field evalua- 
tion, th<? educational outcoDes, and instructional improvc- 
incnts. 

The progran field evaluation variable is assessed by 
sanpling graduates of a program and testing than to deter- 
mine their level of prograr^ specific performance. If the 
individual5 tested demoniitrate knowledge exceeding that of 
graduates of similar programs at comparable institutions, or 
if knowledge of program graduates has improved since a pre- 
vioaT assessment, the institution is eligible for points. 
The test used to determine performance may be an externally 
validated instrunent or it may be an instrument developed by 
the institution itseJf. 

The assessment of educational outcomes is undertaken by 
measuring students' gains in general education or by docu- 
menting the job placement rates of technical institute gra- 
duates. Institutions may only use the ACT-COMP objective or 
composite measures for the assessment of general education 
gains.' Job placement rates must be documented by follow-up 
studies that survey the employment status of technical pro- 
gram gr^uates within 90 days of graduation. 

Instructional improvements that are based on outcome 
Information can also be used as evide?ioe to win funding 
points for institutions. In this case the outcome informa- 
tion consists of opinions surveyed from formerly enrolled 
students or community members and enployers. The 
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institution is eligibie for points v*ien it indicates how the 
survey informatioh is being used to make specific instruc- 
tional iinpcovements . 

Program administrators state that the institutions have 
become increasingly supportive of performance measures in 
the distribution of state appropriations. However, both ' 
proRr.Tii and institution administrators have voiced concern 
over the rising costs involved in outcome evaluation,, If the 
progran is to be successful in the long run there must be 
assurnnce that the process will generate significantly more 
money than it costs. 

Ohly certain segments of the NCHEMS/Kellogg progran and 
the Tennessee pi v. jtjct deal specifically with community col- 
lege concerns. One outcomes project that is entirely a com- 
munity college undertaking is the Program Performance Pro- 
file being developed ty the Office of Institutional Research 
of the Nassau Community College in Garden City, New York. 
The Profile is an outcomes-oriented look at how degree- 
grantin?^ programs are performing. Descriptive statistics 
are gathered o*"! ten criterion variables or "indicators." 
The first two indicators are actually student input vari- 
dbles, involving data on new enrollments in prograns and on 
the academic potential of program students. The remaining 
tfight indicators depict such things as academic achievement, 
persistence, and satisfaction for current -progrjn students 
and, in some instances, for program graduates. TTiese data 

62 



Workins Papor Ho. 10 

provide the basis for the internal review of academic pro- 
grans, the goi»l of which Is to "enable acadenic deparUnents 
to modify and ' improve their curricular offering? in an 
informed manner ." 

Efforts to design and implement standardized tests of 
iearner outcomes are also continuing to evolve. Learner 
outcomes have traditional! ly been assessed through the use of 
5uch standardized measures as the Educational Testing 

>r vice's Undergraduate Assessment Program (UAP) or the Col- 
lege Level Examination Progran (CLEP). These measures were 
designed primarily to determine a student's mastery of 
specific knowledge in given areas. More recent developments 
in learner outcome assesfjnent have emphasized more than just 
content mastery. For example, in 1976 the American College 
Testing Program (ACT) organized the College (Xjtcome Measures 
Project (COMP) with the intent of developing measures that 
ansess students' abilities to use knowledge in out-of-class 
contexts. 

l!!* College Outcome Measure* P.C^iiS.^ 

liner e are three components of the. COMP battery. The 
first two are the Composite Examination and the Objective 
Test. Both aim at cx«nining six areas of general knowledge 
and analysis, three of which are called "pj-ocess areas" 'and 
threp of wtiicn arc called "content areas." The processes 
evalunted are communication, problem solving, and clarifica- 
tion of values. The content areas of interest are the arts, 
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science and technology, a«xi functioning within social insti- 
tutions. Even in the content areas the emphasis is on the 
ability to mobilize factual knowledge in order to address" 
adult-world situation and problems. The test situations ^^re 
created by using advertisements, art prints, television^ . 
docunentaries, newscasts, and other stimuli. j 

In the format of the Composite Exanination, student 
response to these stimuli takes the fom of short written 
answers, longer expository writing, answering multiple 
choice questions, and giving oral responses which are 
audio-taped. ITie exan can be administered to groups and 
requires about four hours to complete. The large anount of 
qualitative information gathered by the exan demands consid- 
erable time for its evaluation. It is estimated that four 
trained faulty manbers can evaluate a student's responses 
in about -jO minutes. The Objective Test is available when 
the institution does not wish to commit the resources 
required for the Composite Examination. It provides the 
sane process and content assessment but accomplishes it by 
allowing the stuiicnt to respond to the test stimuli by 
y,3wering multiple choice questions, "me Objective Test is 
something of a short- ^«nd version of the Composite Exanina- 
tion that does not gather the richer qualitative data of the 
Com^«site Exanination. but is simple and inexpensive to 
score. Ty.e third comj«nent of the CCMP battery is the 
Activity Inventory. The Inventory allows students to report 
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on r.hPir own involvement in out-of-olass activities. The 
format consist.-i of niultiplCrchoice itens in COMP's six out- 

\ 
\ 

come zreas. Students select ajnong responses that indicate 
low to high levels of participation in activities within 
those areas. By docinenting the student's life experiences, 
the inventory is designed to iound out the profile provided 
by the other COMP measures. 

These fresh approaches to the problems of outcome meas- 
urement and their applications indicate that interest in 
outcome evaluation is alive and well. 
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.'^>ctlon IV - Summary 

v.. 

Drawing together the state-of-thc-drt of outcome 

evaluation in higher education is an eiusive task* This is 
true mainly because attempts to assess outcomes have been 
based on a wide variety of methods and analysis. The 
state-of-the-art has yet to emerge as any one cohesive field 
of investigation. 

The absence of a single state or method of outcome 
evaluation is easy to understand. This paper has pointed 
out that different concepts of outcomes and types of out- 
comes abound. lYie assessment of outcomes also varies 
greatly depr^ndinft on the scope of the research- whether it 
is conducted, at a local, regional, or national level. 

' i 

These different dimensions combine in ways that demand 
different methodological approaches. For exanple, a value- 
added concept of a Type 2 outcome assessed at the national 
level requires a. very different approach than a simple out- 
put description of degrees produced at the local level. In 
the value-addqd approach, both (entry and exit data must be 
g.:itherH in order to control for the value the stur^ent 
' brings to the institution. If the outcome oT interest is an 
affective Type 2 variable, meti-uxlological considerations of 
psychological assessnefit enter the picture. Finally, if the 
data are from a nationwide study, it is usual that sophisti- 
cated procedures, perhaps linear regression, will be util- 
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ized to deterntine whether or not statistically significant 
changes in the controlled psychological variable have 
occurred* This kind of outcome evaluation is obviously 



c<r*'*tificates awarded by an institution. 

This example of two widely differing outcome evalua- 
tions is offered to illustrate the fact that outcome evalua- 
tion is not near to being a single definable entity.- The 
state-of-the-art varies greatly depending on the question at 
hand. A brief review of the types of outcomes will help to 
sunmarrize the various approaches to outcome evaluation. 

Type 1 outcomes, when assessed at the local level, are 
usually approached with methods that rely on descriptive 
statistics or limited survey rcsearnii. Transfer and termi- 
nal degree students are counted, and vocational students are 
surveyed to determent their success of finding work. In 
regional and national Type 1 outcome studies, performance 
comparisons between transfer and native students are made; 
large-scale foUow-up studies are conducted to determine the 
relative worth of community college vocational training; or 
information form existing national survey databases is mani- 
pulated statistically to determine whether or not students 
have benefited materially form their community college 
training. Regional and national studies are usual^ly con- 
structed to compare community college students to other stu- 
dents or to non- students. Local studies of Type 1 ootcomes 
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typically describe outcomes without ircluding a comparison 
group. 

Type 2 outcome studies are more diverse in fchelr 
methods than any other type. Vne assessment of student 
attitixJcs may be approached ualnj 3 simple Ukert scale. On 
the other exirmn, subtle and sophistic^etl experimental 
designs may be used to test students for changes in abstract 
rearoning, self-concept, or fbr other psychological changes. 
As in the case of Type 1 studies, the research methodology 
may or may not control for input variables, and may or may 
not include comparison groups. The diversity of the cogni- 
tive and affective variables that can be researched as out- 
comes nacessarily nwkes the state-of-the-art a diffuse 
entity. 

Type 3 outcomes, most commonly the local econonic and 
codimunity service impacts of an institution, arc virtually 
always assessed using Institutional records and survey 
research methods. As has been discussed, the limitations to 
these studies are the limits of survey research in general. 
Ihe most thoroMgh studies sample and survey important com- 
munity subpopul ations such as business and industry, the 
retired, and local government. These studies also include 
follow-up on non- respondents conducted by trained personnel . 
The least thorough studies arc those that are based on a 
general coBWunity mai I ing and simply report the obtained 
responses. 
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Type U outcomes are the abstract social benefits that 
camLTity colleges offer. Because these outcome are not 
anendble to empirical research, the state-of-the-art of 
their assessment is not a matter of method. Rather, it is a 
matter of analytical insight grounded in institutional and 
social research. For exaraple, writers may attempt to assess 
the corar-LTiity coUcBCs' role in promoting social mobility 
and economic egalitarian ism. For this, they must draw on 
the large body of research that depicts the employment and 
income picture for community college graduates as well as 
consider the basic employment and mobility picture of the 
country as a whole. In the absence of good aggregated 
research, the understanding of Type 4 outcomes cannot be 
advanced. In this way, the assessment of Type n outccmes is 
dependent on *.he research of other impacts of education. 

On the whole, outcome evaluation will remain a diverse 
)king. Th^^^^^Taste^^ attempts to incorporate outcone 
data into institutional planning indicate that interest in 
outcomes is continuing to grow and continuing to diversify. 
As institutions face greater accountability in challenging 
financial times, outcome assessment will became more impor- 
tant and, perhaps, even more eclectic in its methods. 
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Appendix A 



ASSESSMENT/PUCEMENT MODEL 
The AsitMrnent/ Placement Syircm 
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The Learning, Assessxnent, and Retention Consortium (LARC), consisting 
of 15 colleges in Northern California, has published a Progr am Guide 
(1982) which indicates, in the model above, that one of the major 
objectives of the consortium is the assessment of learning outcomes 
in the remedial programs offered to students in the colleges of the 
consortium. 
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MARYLAND COMMUNITY COLLEGES 
STUDENT FOLLOW UP QUESTIONNAIRE 

FIRST-TIME STUD ts, FALL 1976 

TWt pu-pote o( quethornoirt it to Kelp your communii, colli ,v ond ihe Siole Board (or Community College* oueir onj 
impro.« ih.ir progfom* Pl.ot. cofnpl.ie il promptly (evtn .1 you look only one or l-o <ou.»e») ond return ,n the envelope pro 
«id»d All oniwer* will be Hfictly tonfidentioi TKonk you loi yowt otiittonce. 
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A. Pttaie chfclt what yo*.' hgpfd lo ochitvc al tKii com- 
munity collf QC 
P 1 TqVc C'>ufl«l wilJifjul wurbtn^} Toward O 
degree or ceri'^icole 

□ ? Ctrnfiroie 
Q 3 Ai^ocioic degree 

B Flff«je 'hecV the one ^Jotemenr wh»cH moit ctotcly 
CDffaf on<b yo^ir prmcv^ r«ii»*<^ for a)t«idkf«f 1^ 
CoUcfc 

Q 1 fiplornhon ol new career or aro^fniic oreo* 
P 7 P'rprjroiion for immcdioie f"try lOiTo 0 career 
Q 3 PrepOtoliOn ♦or Irornfer to a I'^ur ypor miliUiliOn 
Q 4 Updote ikilU lo' CI job currenlly held 
Q 5 IniereH ond lelf ertricKment 

□ 6 Othr-r (tpeC.Jy) 

C Wot your yool (ind'C'»'ed in lifm B) ochieved by iKe 
time /ou iefl thtt community college? 

□ 1 Yei 

□ ? No 

D D«d y')t> cjtlrnfj \h\\ fommunily college piimorily on O 

pott I. TP o' full tirn* bani'^ 
Q 1 PoH (I I (fcd'U or If per trfm) 
Q V full *<me (12 C'fdiU or M^orp prf Irrm) 

[. How totitficd were you with the quality of clai^room 

inilruclion in your progrom of \tudy'^ (checli one) 
[J 1. Enlremely laliifntd 
[') 2 Sotilfred 
n 3 Un»aliif.«d 

f Mow ntiifpd were you with tht ovffoll quolily o< »hii ' 

cofnrpunity tollege? (checi* one) 
rj 1 Fiirrmely lolit^ed 
( ] 1 Sntiifted 

[j 3 Un|tlti|fi«d 
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C. Plfoie reipond to thtt item if you .ore no longer o 
itudeni ot thu coltpgr 

Itjted below ore tome ocodemic. employment, fmon- 
ciol. ond penonol reotoni why o »tudent might Ipove 
coHegt. To whol extent were thete your rtoiont for 
leaving lhi« college? (Check o\ mony o» opply ) 

43 Q o Achlevfci fducotional gool 

44 Q b CKonged educotionol goal 

43 Q c SchfdwMng conflict Oe»>reen \0b end iK>d«Cj 

46 Q d Accepted o job 

47 Q « W«nl into mililory lervtce 

0 Progrom or cogrtei not ovaiioble at thit colltgt 

4f Q g. Dt««ati|foct*on with progrom 

50 Q K Unture about my cKoice of mojor 

3* Q Count wofb nol cKoHengmg 

D i grodei 

33 k, l*ound couriei too difficult 

*4 Q I Dtunttified with quality of teochmg 

25 Q m Tronilerred 

M Q n Applied, but could nol obtoin finoncol Jtj 

57 Q o f inonciol otd woi no! iv/fTicient 

^' D P Child core loo co»tly 

3^ D ^ Thiv college wa» too expeni»ve 

^ Q ' Per^onol/morrioge 

H tf you ore no longer o itudeni at thit coHegr. look ol 
•he obove lut and lelect the three '"Oit importont 
rfoioni why you did not rtturn to thi> college (l<tl. 
in order of imporfonce. the opprop'iote tetter [o. b, 
C ric I in the bonei below ) 

*1 Fint Q 

47 Second □ CLEAniMnHOUSE:F=OI?. 

*i □ JUrilOrUvOLLtGES , 

UNIVERSITY OF CALIFORNIA 

DEC 7 1984 
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